Run every analysis job as a trial - #1122

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

trials.experiment_id is NOT NULL. Analysis trials borrow the task's first
live experiment membership, falling back to any live trial's experiment.
The analyzer host task gets a dedicated experiment so report trials never
land in a user's. The experiment-page trial loader now filters kind='agent',
since analysis trials carry experiment ids after this.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Analysis trials need an experiment. Putting them in the experiment they
grade polluted every experiment-scoped surface, so each experiment now
gets one shadow: a hidden experiment named '<name> (qa report)' that
holds the qa and audit trials for its tasks. The experiment page links
both directions, the task page lists the shadow like any membership,
and the QA page lists all reports. Analyzer report trials keep their
own host experiment.

QA results now carry _graded_by (the qa trial id) so the UI can jump
from a graded trial to the trial that graded it, and the analysis
trials get their own drawer group on the task page.

Also: a Daytona teardown that finds the sandbox already gone is a
success, not an error, and the post-trial hooks are guarded so a hook
failure can never wedge a settled job -- the cleanup sweep re-runs
stage advancement.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

The dashboard rows now carry the report id, so each experiment links to
its qa report in place. The QA-page report listing goes away -- one way
to get there, not two.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

The qa cost views read the analysis_costs ledger, which the deleted
block framework wrote. QA and audit now run as trials, so their spend
sits on the trial row. Task scope gains a third union branch over
analysis trials; experiment scope counts member tasks' analysis trials,
owned when they live in this experiment's qa-report shadow. Legacy
ledger rows still count.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Preview QA/audit trials were 400ing on the shared Anthropic/Bedrock spend
cap. Point the analysis-model default at fireworks/glm-5p2 on preview apps
(oddish-pr-*) only, gated like _DEFAULT_JIT_ROLE. Prod keeps claude-sonnet-4-6
and ODDISH_ANALYSIS_MODEL still overrides. fireworks/glm-5p2 routes to the
Fireworks endpoint, not the Bedrock chokepoint, so it maps cleanly at pickup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

No new migrations (single head holds). Take staging's #1160 version of the
migration-head test (their official fix supersedes this branch's stopgap).
Deleted trajectory/analyzer files stay deleted. The preview Fireworks-GLM
analysis default is unaffected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

QA is task-scoped: the verdict grades every live trial of the task,
across experiments, as of the run that produced it. The experiment page
now carries that honestly:

- The per-task verdict chip opens the task overview (the existing
  cross-experiment QA surface with 'elsewhere' trial chips) instead of
  being display-only. No new panel; the overview already answers 'which
  trials did this verdict grade'.
- When some of the viewed experiment's settled agent trials carry no
  grade, the verdict came from a run that did not include them; the chip
  renders dashed with an 'earlier run' tooltip and the ungraded count,
  so a verdict from elsewhere is never mistaken for one over these rows.
- The '(qa report)' shadow experiment is demoted to a debug surface:
  the header hop links and the dashboard chip are admin-only
  (isOrgAdminRole), matching how the verdict itself is already inline
  everywhere a non-admin needs it.
- On the shadow page, qa/audit trials get their own labeled columns
  ('QA run', 'Pre-trial audit') instead of grouping under the agent
  name — the page reads as the QA machinery, not a mirror of the graded
  experiment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

…rison

Staging added 24 commits, 5 migrations, and an EC2/execution-lane worker
path. Resolution notes:

- worker_job_single_job.py: staging's execution-lane + sandbox-capacity
  filters and this branch's retired-kind guard both claimed SQL $6. Kept
  both; the kind filter is renumbered to $10 behind staging's $6-$9.
- cleanup.py / config.py / backend worker functions.py were mixed hunks:
  staging's EC2 imports, settings, secrets and teardown function are kept;
  the Daytona/analyzer-block settings and imports this branch removed are
  dropped.
- Migrations re-chained onto staging's new head (task_browse_summary_002),
  single head restored. Replayed on a scratch DB: from staging's head,
  'upgrade head' runs this branch's chain and creates trials.kind and
  experiments.shadow_of while staging's sandbox_capacity_leases and
  worker_jobs.execution_lane survive.

The important find: staging shipped the cohort comparison (#1178/#1189) on
top of analyzer_blocks -- it caches its computed comparison there and reads
legacy trajectory-summary rows as a fallback behind the
trials.trajectory_summary mirror this branch's QA importer feeds. Dropping
the table would have deleted a live feature's storage, so drop_analyzers_001
now drops only analyzers and analyzer_experiments, AnalyzerBlockModel is
retained, and the cohort service carries the two type constants it needs
locally instead of importing the deleted block framework.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

The cohort comparison was the only thing still standing on the analyzer
block framework, and its generation path ran through it -- so keeping
analyzer_blocks alive only produced a card that 500s. Remove the feature
whole: the block package, the cohort service, prompts, taxonomy, route,
frontend section and types, and its tests. drop_analyzers_001 drops
analyzer_blocks again, AnalyzerBlockModel is gone, and the preview seed no
longer copies blocks. Re-running the comparison as a qa-kind trial is the
natural follow-up.

Also fixes the trial card lying about QA state: the rerun creates one
task-level qa trial and never stamps this trial's analysis_status, so the
card showed 'No analysis yet' while the run was live. It now reads the
task's QA state, says 'QA is running', and links to the qa trial doing the
grading.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

The Findings header appended the pre-trial audit's dollar cost next to the
finding count, putting a billing figure beside a quality result. QA spend is
deliberately excluded from cost tiles and per-row annotations everywhere
else, so this was the one surface breaking that rule; the audit's spend still
shows in the task QA cost rollup. The now-unused checksCostUsd prop and its
pass site go with it.

The 'graded by' link resolved the qa trial by searching the host's ordered
trial list, but the qa trial lives in the shadow experiment and is usually
absent from it -- so the button either never rendered or silently did
nothing. It now navigates in place when the trial is present and deep-links
/tasks/<id>?trial=<qa id> otherwise.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Staging added two more cohort commits (#1209, #1196) while this branch was
in flight. The cohort comparison is superseded by running analysis as trials,
so the removal stands: every cohort file staging modified stays deleted, and
the cohort types come out of the frontend type module again. Also drops a
stale reference to cohort blocks from the qa_cost attribution note.

Single alembic head, imports and typecheck clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Comment-only. The worst offenders were explaining a one-line change in five
lines (the preview secret publish) or restating the diff (the analysis-model
default, the legacy WorkerJobKind members, the verdict-keeping append path,
the QA cancel path, and the analysis image/verifier notes). Each keeps the
non-obvious reason and drops the elaboration.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Analysis (QA + audit) now defaults to fireworks/glm-5p2 unconditionally
instead of Claude via Bedrock, replacing the preview-only split added
earlier this session. Routes to the Fireworks provider, which already
carries FIREWORKS_API_KEY for agent trials on the same path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Remove every analysis opt-in: tasks always run QA when their agent
trials settle, and every task version gets its pre-trial audit at sweep
time. The --run-analysis / --enable-analysis flags, the run_analysis /
enable_analysis API fields, the per-org pre-trial setting (endpoints,
schemas, worker gate, dashboard route), the ODDISH_PRE_TRIAL_ENABLED
flag, and backend/worker/audit_opt_in.py are gone. The
tasks.run_analysis column stays, always true, for old readers.

The task verdict now needs evidence before it is asked for: at least 5
QA-eligible trials from at least 3 distinct agents. Below the bar the
QA brief tells the model to omit the verdict, the importer treats the
missing verdict as expected, and the task completes without one instead
of judging off a handful of runs.

Tests that assumed audit-free tasks (trial counts, {task_id}-N index
layouts, agent-keyed job maps) now filter on trials.kind or compute the
next index instead of hardcoding it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj

Bugs: a below-evidence-bar QA run could never succeed (the brief said
omit the verdict key, the verifier required it -> every run retried to
exhaustion; the brief now asks for "verdict": null), the no-verdict
import path fired the GitHub PR-refresh hook twice, and the analysis
verifier silently skipped JSON validation when its interpreter probe
fell through (now an unconditional python3, which the analysis image
guarantees). The QA brief also gave two conflicting trajectory_summary
shapes and cited a taxonomy label that no longer exists; one shape and
real labels now.

The "start QA or complete the task" sequence lived in four drifting
copies (settlement, legacy advance, backfill, cleanup healer); they now
share one start_qa_for_task helper built on the queue_verdict /
abandon_verdict primitives.

Dead code from the analyzer/opt-in deletions: start_verdict, the
should_store knob on sync_pre_trial_to_task_version, queue_verdict's
error param, notify_analysis_update, PROBE_ANALYZER_MODEL, the
get_analysis_queue_key duplicate, four unused analyze re-exports and
their dataclasses, unused imports, an always-zero cleanup metric pair,
and the leftover run_analysis entry in the sweep append payload. The
frontend's three hand-rolled "is QA live" predicates now share the
job-status helpers. Comments that narrated deleted machinery are
rewritten to describe what exists.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GzxfgtotWcStMXaRAkvotj